The present invention relates to optimizing a network in a distributed big data environment. A network is defined as a distribution of resources across products. For example, the resources may be seats in an airplane cabin, and the products may be airline tickets on flights between various destinations. Similarly, the resources may be cabins within a cruise ship, and the products may be cruise tickets or packages between various destinations. However, the network is not limited to these examples, and may include any situation in which resources are distributed across products having some capacity.
Data describing the resources may be analyzed to determine the optimal distribution of the resources across products to be offered for sale to customers. However, the data often include too many items to be stored and/or analyzed together. For example, in the cruise industry, a product's dimensions may include 1 ship, 1 week, 6 destinations, 9 segments, 10 cabin categories, 2 package types, 12 markets, 3 fare types, 7 price tiers, 3 occupancy types, and 45 weeks, such that the product includes 36,741,600 items. Further, the distribution may be subject to various constraints, such as price rationality rules, scheduling priorities, etc., which complicate the analysis. Accordingly, it would be advantageous to provide a method of handling the very large and complex data analysis that is required to determine the optimal distribution of resources across products.